Developing Population Codes by Minimizing Description Length

The Minimum Description Length principle (MDL) can be used to train the hidden units of a neural network to extract a representation that is cheap to describe but nonetheless allows the input to be reconstructed accurately. We show how MDL can be used to develop highly redundant population codes. Each hidden unit has a location in a lowdimensional implicit space. If the hidden unit activities form a bump of a standard shape in this space, they can be cheaply encoded by the center of this bump. So the weights from the input units to the hidden units in a self-supervised network are trained to make the activities form a standard bump. The coordinates of the hidden units in the implicit space are also learned, thus allowing exibility, as the network develops a discontinuous topography when presented with di erent input classes. Population-coding in a space other than the input enables a network to extract nonlinear higher-order properties of the inputs. Most existing unsupervised learning algorithms can be understood using the Minimum Description Length (MDL) principle (Rissanen, 1989). Given an ensemble of input vectors, the aim of the learning algorithm is to nd a method of coding each input vector that minimizes the total cost, in bits, of communicating the input vectors to a receiver. There are three terms in the total description length: The code-cost is the number of bits required to communicate the code that the algorithm assigns to each input vector. The model-cost is the number of bits required to specify how to reconstruct input vectors from codes (e.g., the hidden-to-output weights in Figure 1). 1Corresponding author. Algorithms and Architectures: Learning Algorithms|Oral